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The  title  for  this  issue  of  the  Navigator  says  it  all-"Future 
Shapes  of  Whafs  to  Come..."  The  NAVO  MSRC  is 
undergoing  a  carefully  planned  series  of  enhancements 
which,  when  completed  in  summer  2000,  will  provide 
one  of  the  most  capable,  productive,  and  balanced  high 
performance  computing  (HPC)  environments  in  the 
world.  These  enhancements  substantially  boost  the 
MSRC's  computational  capabilities  across  the  primary 
HPC  architectures  we  support — distributed-memory  par¬ 
allel,  shared  memory,  and  parallel  vector.  The  enhance¬ 
ments  also  boost  the  mass  storage  and  networking  capa¬ 
bilities  of  the  MSRC  as  well. 

The  most  successful  and  requested  HPC  system  within 
the  NAVO  MSRC,  the  Cray  T3E,  has  been  expanded  by 
33  percent  to  1,088  processors  and  approximately  400 
GB  of  aggregate  memory.  A  new  IBM  RS/6000  SP  sys¬ 
tem  with  1,336  POWER3/375  processors  and  1.3  TB  of 
Siggregaie  memory,  one  of  the  largest  systems  ever  built 
by  IBM,  is  being  installed  as  this  issue  goes  to  press.  Both 
the  Cray  T3E  and  the  IBM  SP  are  intended  to  provide 
the  premiere,  high-end  computational  environment  for 
applications  throughout  Department  of  Defense  (DoD) 
that  simply  cannot  be  run  on  lesser  systems — applica¬ 
tions  that  typically  run  as  DoD  challenge  projects.  To 
supplement  this  enormous  distributed-memory  parallel 


computational  capability,  we  have  installed  a  Sun 
HPC 10000  computational  server  with  64  processors  and 
64  GB  of  shared  memory — a  unique  resource  which  pri¬ 
marily  supports  interactive  shared-memory  HPC  work 
and  diagnosis/debugging  work  for  large  parallel  applica¬ 
tions.  The  SGI  Origin  2000  systems  (now  known  formal¬ 
ly  as  the  SGI  2800)  have  been  merged  and  expanded  to 
a  single  256-processor  system  running  a  single  IRIX 
image,  greatly  enhancing  their  capability  to  support 
shared-memory  Challenge  project  work.  The  Cray  T932 
will  be  retired  at  year's  end,  with  two  additional  Cray 
SVl  system  nodes  (for  a  total  of  four)  being  added  to  the 
existing  Cray  SVl  system.  The  upgraded  SVl  will  be 
configured  with  4  gigawords  (i.e.,  32  GB)  of  shared 
memory — the  largest  memory  ever  offered  on  a  parallel 
vector  system  at  the  NAVO  MSRC.  Finally,  the  internal 
mass  storage  and  networking  capabilities  of  the  MSRC 
have  been  completely  reengineered  for  performance  and 
resiliency,  providing  a  perfect  fit  with  our  new  OC-12 
wide-area  Defense  Research  and  Engineering  Network 
(DREN)  connectivity. 

We  invite  you,  the  DoD  user  community,  to  let  us  assist 
you  in  utilizing  this  cutting-edge  capability  in  support  of 
your  HPC  needs. 


About  the  Cover: 

Future  shapes  of  what’s  to  come. ..The  US  Army’s  Theater  High  Altitude  Area  Defense  Interceptor  (THAAD)  is  a  finless  mis¬ 
sile  that  utilizes  ten  liquid  bi-propellant  divert  and  attitude  control  jets  for  maneuvering  during  acquisition  and  homing  of  an 
incoming  target.  Through  extensive  use  of  Computational  Fluid  Dynamics  (CFD),  researchers  have  been  able  to  generate 
engineering  and  aerodynamic  data  that  would  otherwise  not  have  been  available  for  the  development  and  checkout  of  the 
THAAD  missile  (see  story  on  page  12). 
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Major  Shared  Resource  Center  (MSRC): 
Delivering  Science  to  the  Warfighter 
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The  NAVO  MSRC  provides  Department  of  Defense 
(DoD)  scientists  and  engineers  with  high  perform¬ 
ance  computing  (HPC)  resources,  including  leading 
edge  computational  systems,  large-scale  data  stor¬ 
age  and  archiving,  scientific  visualization  resources 
and  training,  and  expertise  in  specific  computational 
technology  areas  (CTAs).  These  CTAs  include  Com¬ 
putational  Fluid  Dynamics  (CFD),  Climate/Weather/ 
Ocean  Modeling  and  Simulation  (CWO),  Environ¬ 
mental  Quality  Modeling  and  Simulation  (EQM), 
Computational  Electromagnetics  and  Acoustics 
(CEA),  and  Signal/lmage  Processing  (SIP). 
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This  project  is  aimed  at  developing  an 
ocean  prediction  system  that  will  revo¬ 
lutionize  the  ability  to  nowcast  and 
forecast  the  global  ocean  circulation  down  to 
the  scale  of  oceanic  fronts  and  eddies.  A 
real-time  demonstration  for  the  Pacific 
Ocean  is  currently  mnning  in  real  time  at 
NAVO  MSRC  with  1/16°  resolution,  a 
demonstration  that  will  be  expanded  to  the 
global  ocean  in  the  near  future.  Department 
of  Defense  (DoD)  High  Performance  Com¬ 
puting  (HPC)  resources  and  DoD  HPC 
Challenge  projects  at  NAVO  MSRC  and 
U.S.  Army  Engineer  Research  and  Develop¬ 
ment  Center  (ERDC)  MSRCs  have  been  crit¬ 
ical  to  this  effort. 

INTRODUCTION 

DoD  HPC  and  a  DoD  HPC  Challenge  grant 
on  the  NAVO  MSRC  SGI/Cray  T3E  are  criti¬ 
cal  components  of  a  coordinated  6. 1-6.4 
Naval  Research  Laboratory  (NRL)  effort  on 
the  "Grand  Challenge"  problem  of  eddy¬ 
resolving  global  ocean  modeling  and  predic¬ 
tion.  The  goal  is  a  fully  eddy-resolving,  data 
assimilative  global  ocean  prediction  system 
with  at  least  1/16°  (—7.5  km)  resolution  in 
the  near  term,  with  an  upgrade  to  1/32° 
when  the  available  computer  power  is  suffi¬ 
cient.  A  1/16°  system  is  currently  running  in 
demonstration  mode  for  the  Pacific  north  of 
20°S,  and  expansion  to  the  global  ocean  is 
planned  in  the  near  future  using  DoD  HPC 
Challenge  resources  at  the  NAVO  MSRC 
and  the  NRL  Layered  Ocean  Model 
(NLOM). 

The  1/16°  Pacific  NLOM  system  already 
gives  a  real-time  view  of  the  ocean  down  to 
the  50-200  km  scale  of  ocean  eddies  and 
the  meandering  of  ocean  currents  and 
fronts,  a  view  with  unprecedented  resolution 
and  clarity.  This  can  be  seen  at  the  URL. 

The  system  has  demonstrated  forecast  skill 
for  a  month  or  more  for  many  ocean  fea¬ 
tures,  including  the  fronts  and  eddies.  The 
assimilation  of  satellite  altimeter  data  into  this 
system  makes  effective  use  of  the  near-real¬ 
time  altimeter  data  from  TOPEX/POSEI- 
DON  and  ERS-2  that  is  available  from 
Naval  Oceanographic  Office’s  (NAVO- 
CEANO's)  Altimetry  Data  Fusion  Center 
(ADFC).  The  effectiveness  of  the  model 


assimilation  of  altimeter  data  is  greatly 
enhanced  by  the  1/16°  resolution  of  the 
Pacific  system,  as  demonstrated  by  compari¬ 
son  to  corresponding  results  at  coarser  reso¬ 
lution.  Other  data,  such  as  sea  surface  tem¬ 
perature  and  sparse  vertical  profiles  of  tem¬ 
perature  and  salinity,  will  be  assimilated  as 
well. 

BACKGROUND 

Ocean  forecasting  is  in  principle  similar  to 
atmospheric  forecasting,  but  with  two  major 
complications:  (a)  ocean  eddies,  at  about 
100  km  across,  are  typically  20  to  30  times 
smaller  than  comparable  atmospheric  highs 
and  lows  which  means  that  roughly  four 
orders  of  magnitude  more  computer  time 
and  three  orders  of  magnitude  more  com¬ 
puter  memory  are  required;  and  (b)  there 
are  relatively  few  observations  below  the 
ocean  surface,  so  data  assimilation  is  effec¬ 
tively  confined  to  using  satellite  observations 
of  the  surface.  The  duration  of  forecast  skill 
for  the  ocean  is  not  restricted  to  the  10-  to 
14-day  limit  for  atmospheric  highs  and  lows. 
We  have  demonstrated  at  least  30-day  pre¬ 
dictive  skill  for  ocean  eddies  and  the  mean¬ 
dering  of  ocean  currents  and  fronts,  given 
sufficient  ocean  model  resolution  and  satel¬ 
lite  altimeter  data  from  TOPEX/POSEIDON 
and  ERS-2. 

A  major  component  of  NRL's  ocean  model¬ 
ing  program  has  been  a  detailed  study  of  the 
resolution  required  for  ocean  prediction.  We 
have  strong  evidence  that  NLOM  and  other 
ocean  models  (including  all  the  popular 
global  and  basin-scale  ocean  models)  need 
to  use  grid  cells  for  each  prognostic  variable 
that  are  at  most  about  8  km  across  at  mid¬ 
latitudes.  Our  research  has  shown  that  dou¬ 
bling  the  resolution  to  4  km  per  cell  gives 
substantial  improvement  but  doubling  again 
to  2  km  gives  only  modest  additional 
improvement.  Due  to  ocean  modeler  pref¬ 
erence  and  choice  of  finite  difference  grid 
design,  there  is  significant  variation  in  how 
such  resolution  is  expressed  in  degrees,  the 
most  common  way  to  describe  ocean  model 
resolution.  For  the  NLOM  grid  it  translates 
to  1/16°,  1/32°,  and  1/64°  resolution,  respec¬ 
tively.  This  is  for  the  global  and  basin-scale. 
Coastal  models  would  use  the  global  forecast 
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for  boundary  conditions  and  would  require 
much  smaller  cells,  but  would  cover  only  a 
limited  region. 

At  4  km,  the  optimal  resolution  is  finer  than 
might  be  expected  based  on  the  size  of 
eddies.  In  relation  to  ocean  eddy  size,  it  is 
similar  to  the  resolution  currently  used  by 
the  leading  weather  forecasting  models  in 
relation  to  the  size  of  atmospheric  highs  and 
lows.  More  specifically,  our  research  has 
shown  that  fine  resolution  of  the  ocean 
eddy  scale  is  required  to  obtain  coupling 
between  upper  ocean  currents  and  seafloor 
topography  via  turbulent  flow  instabilities. 
This  coupling  can  strongly  affect  the  path¬ 
ways  of  upper  ocean  currents  and  fronts, 
including  the  Gulf  Stream  in  the  Atlantic 
and  the  Kuroshio  in  the  Pacific.  The  high 
resolution  is  also  required  to  obtain  sharp 
fronts  that  span  major  ocean  basins  and  a 
nonlinear  effect  on  the  large  scale  C-shape 
of  ocean  gyres,  such  as  the  Sargasso  Sea  in 
the  Atlantic. 

TECHNICAL  APPROACH 

As  far  back  as  1989,  the  President's  Office 
of  Science  and  Technology  recognized  glob¬ 
al  ocean  modeling  and  prediction  as  a 
"Grand  Challenge"  problem,  defined  as 
requiring  a  computer  system  capable  of  sus¬ 
taining  at  least  one  trillion  floating  point 
adds  or  multiplies  per  second.  By  taking  a 
multi-faceted  approach  to  cost  minimiza¬ 
tion,  we  are  solving  the  problem  on  systems 
capable  of  only  a  small  percent  of  this  per¬ 
formance.  One  facet  is  experiment 
sequences  that  use  the  largest  cell  size  possi¬ 
ble  and  an  ocean  basin  rather  than  the 
entire  globe  whenever  possible.  This  only 
gets  us  so  far,  since  in  the  end  there  is  no 
substitute  for  small  cells  and  a  global 
domain. 

Another  facet  is  the  use  of  the  NLOM  which 
has  been  specifically  designed  for  eddy¬ 
resolving  global  ocean  prediction.  It  is  tens 
of  times  faster  than  other  ocean  models  in 
computer  time  per  model  year  for  a  given 
horizontal  resolution  and  model  domain. 
NLOM's  performance  is  in  turn  due  to  a 
range  of  design  decisions,  the  most  impor¬ 
tant  of  which  is  the  use  of  isopycnal  (density 
tracking)  layers  in  the  vertical  rather  than 
the  more  usual  fixed  depth  cells.  Density  is 
the  natural  vertical  coordinate  system  for 
the  stratified  ocean,  and  it  allows  seven 
NLOM  layers  to  replace  the  100  or  more 
fixed  levels  that  would  be  needed  at  1/16° 
resolution.  Another  important  advantage 
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Figure  1.  Zoom  on  the  Kuroshio  current  region  south  and  east  of  Japan.  (Top)  Sea  surface  height 
(SSH)  for  15  January  1999,  from  the  NRL  1/16°  Pacific  model  with  assimilation  of  satellite  altimeter 
data  from  TOPEX/POSEIDON  and  ERS-2.  The  altimeter  tracks  with  data  available  for  this  update  cycle 
are  overlain.  (Middle)  Corresponding  SSH  snapshot  from  a  14-day  forecast  initialized  from  1  January 
1999.  (Bottom)  MODAS  1/8°  SST  analysis  from  satellite  IR  imagery.  The  SST  color  bar  is  designed  to 
highlight  the  Kuroshio  pathway. 


of  this  approach  is  that  there  is  less  need  to 
increase  the  number  of  density  tracking  lay¬ 
ers  as  the  horizontal  cell  size  is  reduced. 

With  NLOM,  halving  the  cell  size  requires 
about  8  times  as  much  computer  power  (4 
times  from  the  number  of  cells  plus  2  times 
from  the  required  smaller  time  step),  but 
with  fixed-level  ocean  models,  the  number 
of  cells  in  the  vertical  should  also  be  dou¬ 
bled,  requiring  about  16  times  as  much 
computer  power. 

A  third  facet  is  a  widely  portable  NLOM 
computer  code,  developed  under  the  Com¬ 
mon  High  Performance  Computing  Soft¬ 
ware  Support  Initiative  (CHSSl)  program, 
that  targets  large  scalable  computers  with 
high-speed  network  connections.  NLOM 
exhibits  very  good  scalability  (wall  time 
speedup  as  more  processors  are  used)  on 
such  systems.  For  example,  we  have  rou¬ 
tinely  mn  NLOM  on  up  to  1,152  Cray  T3E 
processors  at  up  to  a  sustained  speed  of 
about  100  billion  useful  floating-point  oper¬ 
ations  per  second. 

A  final  facet  of  our  efficiency  drive  is  the  use 
of  an  inexpensive  data  assimilation  scheme 
backed  by  a  statistical  technique  for  relating 


surface  data  to  subsurface  fields.  The  statis¬ 
tics  are  from  an  atmospherically  forced  20- 
year  inter-annual  simulation  of  the  same 
ocean  model,  an  application  that  requires  a 
model  with  high  simulation  skill. 

So  far  it  has  been  possible  to  mn  NLOM  in 
demonstration  mode  with  1/32°  resolution 
globally  (72°S-65°N)  and  1/64°  resolution 
over  the  basin-scale  subtropical  Atlantic 
(9°N-51°N),  including  the  Caribbean  and 
Gulf  of  Mexico.  While,  at  present,  these 
require  greater  computer  resources  than 
practical  for  an  operational  product,  they  do 
give  information  on  the  value  added  of 
increasing  resolution  and  insight  into  model 
performance  at  1/16°  resolution. 

RESULTS 

In  August  1999,  we  started  running  1/16° 
Fbcific  NLOM  in  near-real  time  (i.e.,  updat¬ 
ed  every  few  days).  In  hindcast  studies  that 
followed  standard  nowcast  and  forecast 
procedures,  but  used  data  from  a  previous 
time  period,  we  compared  1/16°  Pacific 
NLOM  with  1/4°  global  NLOM.  Both  stud¬ 
ies  assimilated  satellite  altimeter  data  from 
TOPEX/POSEIDON  and  ERS-2,  and  then 
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month-long  forecasts  started  from  the  data 
assimilative  nowcast  states  were  performed. 
An  example  is  shown  in  Figure  1  with  com¬ 
parison  to  an  independent  1/8°  SST  analy¬ 
sis  from  the  Modular  Ocean  Data  Assimil- 
tion  System  (MODAS)  that  is  operational  at 
NAVOCEANO.  The  SST  analysis  shows 
the  correspondence  between  the  Kuroshio 
pathway  and  some  of  the  eddies  seen  in 
the  sea  surface  height  field  from  the  Pacific 
Ocean  model,  a  field  observed  by  satellite 
altimetry  as  shown  by  overlain  ground 
tracks  in  Figure  1  (top  panel). 

The  1/16°  Pacific  model  has  given  the  very 
first  basin-wide  skillful  forecast  demonstra¬ 
tion  of  oceanic  fronts  and  eddies  for  any 
ocean  basin.  The  Pacific  results  also  demon¬ 
strate  that  altimeter  data  alone  are  sufficient 
to  produce  an  accurate  nowcast  when  a 
high  resolution  ocean  model  is  in  the  loop 
to  fill  in  the  space-time  gaps  in  the  altimeter 
data.  The  1/4°  global  model  was  much  less 
successful  in  this  feasibility  demonstration, 
as  was  expected  from  the  earlier  discussion 
of  resolution  requirements.  In  addition,  the 
nowcast/forecast  results  at  1/16°  resolution 
were  substantially  better  than  expected  for  a 
system  that  uses  satellite  altimeter  data  as 
the  only  observing  system,  due  mainly  to 
doubts  about  the  space-time  resolution  ade¬ 
quacy  of  the  altimeter  data.  SST  assimila¬ 
tion  will  be  added  in  the  near  future. 

The  global  results  for  SST  have  also  exceed¬ 
ed  expectations,  particularly  for  a  model 
with  only  seven  layers  in  the  vertical.  The 
embedded  mixed  layer  in  NLOM  gives 
accurate  SST  based  on  accurate  atmos¬ 
pheric  forcing  even  with  no  assimilation  of 
SST  data  (or  altimeter  data).  With  climato¬ 
logical  atmospheric  forcing,  global  NLOM 
gives  SSTs  accurate  to  within  .5°C  for  the 
annual  mean  and  .7°C  for  the  seasonal 
cycle.  Global  NLOM  at  1/2°  and  1/8°  reso¬ 
lution  was  mn  1979-98  with  6-12  hourly 
atmospheric  forcing  from  ECMWF  and  no 
assimilation  of  SST  data,  and  then  com¬ 
pared  to  337  year-long  daily  time  series  of 
observed  SST  around  the  world  over  the 
1980-98  time  frame.  The  median  rms  error 
was  .8  to  .9°C  and  the  median  correlation 
coefficient  was  about  .9,  again  with  no 
assimilation  of  SST  data.  The  modal  bin 
for  rms  error  was  .6  to  .8°C,  and  the  modal 
bin  for  correlation  was  .95  to  1.0. 

Figure  2  shows  similar  results  from  1/16° 
Pacific  NLOM,  but  with  a  comparison 
between  results  using  Fleet  Numerical 
Meteorology  and  Oceanography  Center 
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(FNMOC)  Navy  Operational  Global  Atmos¬ 
pheric  Prediction  System  (NOGAPS)  and 
European  Centre  for  Medium-Range 
Weather  Forecast  (ECMWF)  winds  over  the 
time  frame  1990-1998.  In  both  cases 
ECMWF  thermal  fields  were  used  because 
such  fields  are  not  available  from  FNMOC 
prior  to  late  1997.  However,  the  winds  play 
a  significant  role  in  thermal  forcing,  and  the 
FNMOC  winds  were  used  for  that  purpose 
in  the  FNMOC  forced  run.  As  before,  there 
was  no  assimilation  of  SST  data.  In  both 
cases  the  median  rms  SST  error  was  .84°C, 
but  outside  the  equatorial  region  the  SST 
results  using  FNMOC  were  noticeably  bet¬ 
ter,  .76°C  vs.  .84°C,  using  ECMWF;  with 
median  correlation  of  .96  and  .95,  respec¬ 
tively.  Note  the  three  years  of  daily  SST  in 
Figure  2  show  substantial  differences  in  the 
shape  of  the  annual  cycle  as  well  as  shorter 
time-scale  variability  that  is  captured  by  the 
NLOM  SSTs,  an  indication  of  skill  for  both 


the  ocean  model  and  the  atmospheric  forc¬ 
ing. 

These  results  indicate  that  NLOM  SST  is 
sufficiently  accurate  to  be  used  as  a  plat¬ 
form  for  assimilation  of  SST  data  (which 
has  gaps  due  to  cloudiness)  and  for  SST 
forecasting. 
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Daily  SST:  NLOM  vs  buoy  at  43 °N,  ISO'W 
NLOM  with  no  assimiiation  of  SST 


Figure  2.  Daily  observed  SST  from  a  NOAA  buoy  at  43°N,  130°W  (black)  and  modeled  SST  from 
1/16°  Pacific  NLOM  forced  with  6-12  hourly  winds  from  FNMOC  NOGAPS  (red)  and  the  ECWMF 
(blue)  for  1994  (top),  1995  (middle),  and  1997  (bottom).  Thermal  forcing  (except  the  wind  compo¬ 
nent)  is  from  ECMWF  in  both  cases.  NLOM  included  no  assimilation  of  SST  data.  Statistics  include 
root  mean  square  error  (F^S),  mean  error  (ME),  correlation  (R),  and  skill  score  (SS). 
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Large-Scale  Atom  Simulation 

Dr.  Rajiv  K.  Kolia,  Louisiana  State  University 


Novel  materials  that  can  with¬ 
stand  high  temperatures  and 
extreme  environments  are 
generating  considerable  worldwide 
attention.  Such  materials  are 
tremendously  important  for  defense 
technologies.  The  basic  require¬ 
ments  for  designing  materials  that 
have  low  densities,  elevated  melting 
temperatures,  high  oxidation  and 
corrosion  resistance,  the  ability  to 
resist  creep,  and  high  toughness 
encompass  some  of  the  most  chal¬ 
lenging  problems  in  materials  sci¬ 
ence. 

With  this  DoD  Challenge  grant,  we 
have  performed  large-scale  (106-108 
atoms)  molecular  dynamics  (MD) 
simulations  to  investigate  dynamic 
fracture  in  ceramics  and  nanocom¬ 
posites  and  dynamics  of  oxidation  of 
metallic  nanoparticles.  These  simu¬ 
lations  have  been  executed  with 
highly  efficient,  portable  and  scal¬ 
able,  multiresolution  algorithms 
including  the  fast  multipole  method 
for  the  long-range  Coulomb  interac¬ 
tion,  a  dynamic  load-balancing 
scheme  for  mapping  irregular  appli¬ 
cations  on  parallel  machines,  and  a 
fractal-based  compression  scheme 
for  scalable  I/O  and  data  communi¬ 
cation. 


DYNAMIC  FRACTURE  IN 
CERAMICS  AND 
NANOCOMPOSITES 

Molecular  dynamics  simulations  were 
performed  to  investigate  dynamic 
fracture  in  crystalline  SiC  and  GaAs 
at  various  temperatures  using  256 
nodes  of  Cray  T3E  at  the  NAVO 
MSRC.  The  simulations  were  carried 
out  on  a  thin-strip  sample  for  which 
the  mechanical  energy  release  rate, 
G,  can  be  calculated  from  the  knowl¬ 
edge  of  the  applied  strain,  e,  and  the 
value  of  the  stress,  s,  far  ahead  of  the 
crack  tip:  G  =  Wse/2.  In  addition  to 
the  mechanical  energy  release  rate, 
the  crack-tip  velocity  and  local  stress 
distribution  at  various  temperatures 
were  monitored.  Figure  1  shows  the 
results  of  30-million-atom  MD  simu¬ 
lations  of  SiC  at  room  temperature 
and  1,500K.  We  find  that  large 
shear  stresses  close  to  the  crack  tip 
cause  cleavage  fracture  at  room  tem¬ 
perature.  At  elevated  temperatures, 
dissipative  effects  due  to  dislocations, 
micropore  formation  and  coales¬ 
cence,  and  crack  deflection  cause 
stresses  to  spread  out  all  over  the 
system,  thereby  increasing  the  value 
of  G. 

Similar  effects  are  observed  in  our 
100-million  atom  fracture  simulations 
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Figure  1 .  Shear  stress 
distributions  in  30  million 
atom  MD  simulations  of 
SiC  at  300K  and  1500K. 
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of  GaAs.  The  MD  results  also  indi¬ 
cate  that  the  brittle-to-ductile  transi¬ 
tion  temperature  in  GaAs  (~  600K) 
is  close  to  the  experimental  value 
(630-660K).  Figure  2  shows  snap¬ 
shots  of  three  different  crack  fronts  in 
MD  simulations  of  GaAs  at  room 
temperature. 


amorphous  silica  layers  (see  Figure 
3).  Immersive  visualization  of  these 
simulations  reveals  a  rich  diversity  of 
atomistic  processes  including  fiber 
rupture,  frictional  pullout,  and  emis¬ 
sion  of  molecular  fragments,  which 
must  all  be  taken  into  account  in  the 
design  of  tough  ceramic  composites. 


Figure  2.  Snapshots  of  cracks  in  MD  simulations  of  GaAs. 
Crack  surfaces  are  (a)  (110),  (b)  (111),  and  (c)  (001). 


One  of  the  most  promising  materials 
for  high-temperature  structural  appli¬ 
cations  is  a  fiber-reinforced  ceramic 
composite  of  silicon  nitride  host 
embedded  with  fibers  of  silicon  car¬ 
bide.  The  fibers  are  coated  with 
materials  that  form  weak  interfaces 
between  fibers  and  the  matrix.  We 
have  performed  10  million  atom  MD 
simulations  to  investigate  the  effect 
of  interphase  structure  and  residual 
stresses  on  fracture  toughness  in  a 
silicon  nitride  matrix  reinforced  by 
silicon  carbide  fibers  coated  with 


DYNAMICS  OF  OXIDATION 
OF  ALUMINUM 
NANOCLUSTERS 

Dynamical  oxidation  simulations 
were  motivated  by  experiments  that 
reveal  unique  electrical,  optical,  and 
mechanical  properties  of  nanocom¬ 
posites  consisting  of  passivated  A1 
nanoparticles.  This  led  us  to  per¬ 
form  the  first  successful  MD  simula¬ 
tion  of  oxidation  of  an  A1  nanocluster 
using  MD  simulations  based  on  a 
highly  reliable  interaction  scheme 


that  can  successfully  describe  a  wide 
range  of  physical  properties  of  both 
metallic  and  ceramic  systems.  This 
scheme  includes  changes  in  charge 
transfer  as  the  atoms  move  and  is 
thus  capable  of  treating  bond  forma¬ 
tion  and  bond  breakage.  Dynamic 
charge  transfer  gives  rise  to  compu¬ 
tationally  intensive  Coulomb  interac¬ 
tion  which,  for  the  number  of  atoms 
necessary  in  a  realistic  simulation, 
requires  highly  efficient  algorithms 
that  map  well  onto  parallel  architec¬ 
tures.  The  fast  multipole  method 
(FMM)  of  Greengard  and  Rokhlin 
was  used  to  reduce  the  computation¬ 
al  complexity  from  0(N2)  to  0(N) 
for  the  long-range  Coulomb  interac¬ 
tion  with  extensions  for  stress  calcu¬ 
lations.  A  multiple  time-step  algo¬ 
rithm  further  reduced  the  execution 
time  by  an  order  of  magnitude. 

Both  algorithms  map  very  well  onto 
parallel  architectures. 

Dynamic  oxidation  simulations 
reveal  a  rapid  three-step  oxidation 
process  culminating  in  a  stable  oxide 
scale  in  50  ps.  In  the  first  5  ps,  oxy¬ 
gen  molecules  dissociate,  and  the 
oxygen  atoms  first  diffuse  into  octa¬ 
hedral  and  subsequently  into  tetrahe¬ 
dral  sites  in  the  nanoparticle.  In  the 
next  20  ps,  as  the  oxygen  atoms  dif¬ 
fuse  radially  into  and  the  A1  atoms 
diffuse  radially  out  of  the  nanoparti¬ 
cle,  the  fraction  of  six-fold  coordinat- 


Figure  3.  (Left  panel)  Fractured  silicon  nitride  (red)  ceramic  reinforced  with  silica-coated  carbide  fibers 
(yellow).  (Right  panel)  Close-up  of  the  fractured  composite  system.  Small  spheres  represent  silicon 
atoms,  and  large  spheres  represent  nitrogen  (green),  carbon  (magenta),  and  oxygen  (cyan)  atoms. 
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ed  oxygen  atoms  drops  dramatically. 
Concurrently,  there  is  a  significant 
increase  in  the  number  of  O  atoms 
that  form  isolated  clusters  of  corner¬ 
sharing  and  edge-sharing  OA14  tetra- 
hedra.  Between  30  and  35  ps,  clus¬ 
ters  of  OA14  coalesce  to  form  a  neu¬ 
tral,  percolating  tetrahedral  network 
that  impedes  further  intrusion  of  oxy¬ 
gen  atoms  into  and  of  A1  atoms  out 
of  the  nanoparticle  (see  Figure  4). 

As  a  result,  a  stable  oxide  scale  is 
formed.  Structural  analysis  reveals  a 
40  A  thick  amorphous  oxide  scale  on 
the  A1  nanoparticle.  The  thickness 
and  structure  of  the  oxide  scale  are 
in  accordance  with  experimental 
results. 

Currently  we  are  planning  multiscale 
simulations  to  determine  residual 


stresses,  fracture  toughness  and 
hardness,  friction  and  adhesion,  and 
material  degradation  due  to  oxida¬ 
tion  in:  (1)  polydimethylsiloxane 
(PDMS)  on  alumina  and  silica  sur¬ 
faces;  (2)  nanostructured  composites 
consisting  of  aluminum  nanoparticles 
passivated  with  oxygen;  (3) 

A1/A1203,  Al/SiC,  and  Ti/Ti02 
metal/ceramic  interfaces;  and  (4) 
functionalized  atomic-force  micro¬ 
scope  (AFM)  tips  of  silicon  nitride 
and  carbon  nanotubes. 

These  Challenge  applications  require 
a  methodology  that  can  describe 
physical  and  chemical  processes  over 
several  decades  of  length  scales. 
Quantum  mechanical  (QM)  simula¬ 
tions  based  on  the  density  function 
theory  (DFT)  are  performed  in 


regions  where  atomic  bonds  are 
formed  or  broken;  molecular  dynam¬ 
ics  (MD)  simulations  are  carried  out 
in  non-linear  regions  surrounding  the 
QM  region;  and  the  finite-element 
(FE)  approach  with  constitutive  input 
from  QM  or  MD  calculations  is  used 
in  regions  far  away  from  the  process 
zones.  The  QM,  MD,  and  FE 
schemes  are  integrated  with  an 
approach  based  on  control  theory. 
Algorithms  will  be  designed  to  carry 
out  these  hybrid  QM/MD/FE  simula¬ 
tions  in  DoD's  metacomputing  envi¬ 
ronment  with  multiple  parallel 
machines,  mass  storage  devices,  and 
immersive  and  interactive  virtual 
environments  on  a  grid  with  high¬ 
speed  networks. 


Figure  4.  Initial  stage  of  oxidation  of  an  A1  nanoparticle.  Size  distributions  of  OAI4  clusters  at 
20  ps  (left)  and  31  ps  (right)  are  shown.  Clearly,  the  clusters  coalesce  and  percolate  rapidly. 
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Dynamic  Visualizations 

Dr.  Robert  Moorhead,  Director  VAIL,  Mississippi  State  University 

The  scientific  visualization  group  at  Mississippi  State  University's  National  Science  Foundation 
(NSF)  Engineering  Research  Center  (ERC)  for  Computational  Field  Simulation  has  long  been 
associated  with  NAVO  MSRC.  Dr.  Robert  J.  Moorhead  II,  Director  of  ERC's  Visualization,  Analy¬ 
sis,  and  Imaging  Laboratory  (VAIL) (http://www.erc.msstate.edu/Iabs/vail/)  and  his  group  have 
worked  with  NAVO  MSRC  since  1991.  Highlights  of  VAIL's  visualization  work  targeted  at  the 
METOC  community  include  three  packages:  Interactive  Structured  Time-varying  Visualizer  (ISTV) 
(http://www.erc.msstate.edu/research/thrusts/scivi/html/ISTV),  the  EnVis  toolkit,  and  the  CTHRU 
virtual  environment. 


STV 

ISTV  is  an  X/Motif  application  (see  figure 
1)  using  OpenGL  graphics  to  support 
interactive  visualization  of  a  wide  variety 
of  model  data  on  desktop  workstations 
and  larger  machines.  Although  primarily 
targeted  at  geospatial  datasets,  it  can  be 
used  with  any  3D  (static  or  time-varying) 
structured  data.  ISTVs  use  of  logical  grid 
files  (LGFs)  for  data  description  and 
organization  allows  users  to  specify  com¬ 
putational  grid  parameters  implicitly  (in 
the  case  of  regular  latitude/longitude  grids 
commonly  employed  in  METOC  simula¬ 
tions)  without  the  need  to  create  large 
grid  files  to  complement  the  simulation 
solutions. 


The  inherent  size  of  basin-scale  ocean 
models  makes  intelligent  data  handling  a 
cmcial  part  of  the  system.  In  many  cases, 
we  cannot  expect  to  read  the  entire  grid 
or  solution  set  into  main  memory.  Instead, 
ISTV  allows  the  user  to  subsample  large 
datasets  before  reading;  the  user  can  also 
combine  this  with  reading  only  a  selected 
region  of  interest  to  further  limit  the  mem¬ 
ory  requirements  of  the  visualization.  A 
fixed  amount  of  memory  is  used  as  a 
local  buffer  to  enhance  the  interactive 
speed  of  the  system. 

A  calculator  module  is  available  to  com¬ 
pute  derived  values,  as  they  are  needed. 
Many  simple  operations  already  exist  in 
the  calculator,  and  more  complex  or 


Figure  2 


domain-specific  computations  can  be 
added  to  the  code.  When  animating,  data 
are  linearly  interpolated  along  the  time 
axis  from  the  two  adjacent  time  steps.  All 
data  derivation  and  interpolation  opera¬ 
tions  handle  invalid  values  like  "land 
masks"  found  in  many  ocean  model  data 


ISTV  supports  the  simultaneous  visu¬ 
alization  of  model  output  from  several] 
independent  sources,  without  regard 
to  differences  in  the  sampling  fre¬ 
quencies  of  the  various  grids  (see  fig¬ 
ure  2).  This  capability  allows  users  to 
examine  relationships  between  forc¬ 
ing  fields  (e.g.,  wind  data)  and  the 
output  of  ocean  circulation  models. 


Figure  1 


sets. 
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An  important  design  consideration  in 
ISTV  is  the  ability  to  work  with  data  in  its 
native  file  format.  Converting  to  a  new 
format  is  costly  and  problematic,  so  ISTV 
does  not  require  this.  The  code  was 
designed  so  that  a  reader  for  a  new  file 
format  could  be  integrated  with  a  minimal 
amount  of  effort. 

Normally  ISTV  mns  as  a  single  job  on  a 
single  computer.  Quite  often,  however, 
the  data  files  reside  on  a  central  file  server 
rather  than  on  the  visualization  machine. 
In  this  case,  ISTV  has  a  data  server/visual¬ 
ization  client  option  in  which  each  task 
executes  on  the  appropriate  machine;  net¬ 
work  communication  between  the  data 
server  and  front-end  visualization  client  is 
managed  by  message  passing  interface 
(MPI). 

ENVIS  TOOLKIT 

The  EnVis  toolkit  is  built  upon  the  COST 
(Common  Object  Software  Toolkit) 
framework  developed  at  the  ERC  to  pro¬ 
vide  a  common  code  base  for  the  devel¬ 
opment  of  scientific  visualization  tools. 
Initial  development  focused  on  providing 
interactive  visualization  tools,  but  with  the 
rapid  increase  in  model  resolutions,  we 
also  added  batch-mode  visualization 
capabilities. 

Since  the  machines  serving  datasets  often 
contain  no  host-based  graphics  subsys¬ 
tems,  we  have  moved  from  using 
OpenGL-based  rendering  to  either  using 
offscreen-rendering  capabilities  provided 
by  the  Mesa  graphics  library 
http://mesa3d.sourceforge.net  or  other 
software-only  rendering  techniques.  We 
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have  used  our  Mesa-enabled  offscreen 
renderer  on  the  NAVO  MSRC  Origin 
2000  machines  and  are  now  engaged  in 
porting  the  toolkit  to  Solaris  for  use  on 
both  multiprocessor  Sun  machines  as  well 
as  networks  of  Sun  workstations  (see  Fig¬ 
ure  3). 

CTHRU 

CTHRU,  a  virtual  environment  first  devel¬ 
oped  at  the  ERC  for  the  Supercomputing 
1995  conference,  has  been  enhanced 
with  new  features  in  the  past  year.  Origi¬ 
nally  designed  to  demonstrate  the  compu¬ 
tational  steering  of  a  mnning  NLOM 
ocean  model  from  a  visualization  front- 
end  using  a  CAVE,  more  recent  uses  of 
the  system  have  focused  on  support  for 
other  models  (e.g.,  NCOM  Chesapeake 
Bay  circulation  model). 

New  features  include  a  vertical  cutting 
plane,  which  the  user  can  interactively 
place,  and  menu  items  for  changing  level- 
of-detail  subsampling  rate,  isosurface 
threshold,  and  layer  selection  (see  Figures 
4-6).  Since  current  applications  use  host- 
based  datasets,  CTHRU's  performance 
has  also  been  improved  for  mnning  on  a 
single  machine  by  using  the  native  SGI 
MPI  libraries  which  use  shared  memory  as 
opposed  to  sockets  for  data  communica¬ 
tion. 


In  addition  to  development  of  tools,  VAIL 
tries  to  help  the  NAVO  MSRC  Visualiza¬ 
tion  Lab  by  evaluating  visualization  tool¬ 
kits  developed  elsewhere.  Recently  stu¬ 
dents,  some  of  whom  work  in  the  NAVO 
MSRC  Visualization  Lab  when  they  are 
not  in  school,  have  been  evaluating  the 
visualization  of  METOC  data  with  the 
new  open-source  release  of  Data  Explorer 
http://www.opendx.org/  and  VisAD 
http://www.ssec.wisc.edu/~billh/ 
visad.html,  a  Java-based  visualization 
class  library. 

NOTE:  Data  are  courtesy  of  the  Naval 
Research  Laboratory,  and  the  use  thereof 
does  not  constitute  an  endorsement. 
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The  U.S.  Army’s  Theater  High  Altitude  Area  Defense 
Interceptor  (THAAD)  is  a  finless  missile  that  utilizes 
ten  liquid  bi-propellant  divert  and  attitude  control  jets 
for  maneuvering  during  acquisition  and  homing  of  an 
incoming  target.  This  type  of  control  system  gener¬ 
ates  aerodynamic  interactions  between  the  control  jet 
exhaust  and  the  oncoming  flow  that  must  be  included 
in  the  autopilot  design  to  achieve  robust  and  reliable 
flight  performance. 

While  extensive  wind  tunnel  data  have  been  collected 
from  four  tunnel  entries  for  the  development  of 
appropriate  guidance  algorithms,  these  cold  exhaust, 
subscale  jet  interaction  (JI)  data  do  not  always  simu¬ 
late  the  true  aerodynamics  of  an  actual  burning 
thruster  at  all  altitudes. 

Through  extensive  use  of  computational  fluid  dynamics 
(CFD),  however,  researchers  have  been  able  to  gener¬ 
ate  engineering  and  aerodynamic  data  that  would  oth¬ 
erwise  not  have  been  available  for  the  development 
and  checkout  of  the  THAAD  missile.  The  data  have 
been  successfully  utilized  for  improved  design  under¬ 
standing,  autopilot  tuning  and  sensitivity  studies,  ground 
and  flight  test  data  analysis,  and  improved  flight  simu¬ 
lations.  Each  of  the  analyzed  aspects  of  reaction  jet 
control  has  contributed  to  an  improved  understanding 
of  the  missile  aerodynamics  and  performance  objec¬ 
tives,  and  greater  confidence  in  the  present  design  and 
future  requirements  has  been  achieved. 


Encouraged  by  this  success,  THAAD 
is  continuing  to  utilize  HPC  resources 
to  examine  additional  aspects  of  JI 
phenomena  before  going  to  the  next 
round  of  flight  tests. 

Current  research  is  focused  on  the 
possible  effects  of  hot  jet  exhaust  flow 
over  the  infrared  (IR)  seeker  window 
under  certain  flight  conditions.  A 
complete  engineering  design  requires 
anticipation  of  software  algorithms 
that  may  be  necessary  to  acquire  and 


track  dim  targets, 
and  it  is  wise  to  include  any 
effects,  no  matter  how  slight,  that 
may  affect  seeker  performance. 
This  aspect  of  the  JI  analysis  is 
made  more  difficult  by  the  high 
degree  of  computational  grid  res¬ 
olution  needed  over  a  very  large 
spatial  domain  for  accurate  simu¬ 
lation  of  the  expanding  jet  flow. 
Computational  simulations  are 
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currently  being  run  on  NAVO 
MSRC’s  Cray  T-90  and  the  Space 
and  Missile  Defense  Command’s 
Origin  2000  architectures  for  this 
large  memory  problem. 

The  NAVO  MSRC  Scientific  Visual¬ 
ization  Staff  has  worked  closely  with 
the  THAAD  researchers  to  develop 
a  portable  graphics  tool  to  model 
the  new  flow  field  datasets.  This 
tool  helps  reconstruct  the  induced 
flow  separation  at  attack-of-attack 
in  an  immersive  environment 


to  illustrate  the  various 
complex  aspects  of  reaction  jet 
control.  Flow  visualization  is  a 
key  tool  that  aids  the  researcher 
in  understanding  the  relation¬ 
ship  between  complex  aerody¬ 
namic  phenomena  and  the 
behavior  of  the  missile  on  its 
way  to  target  intercept. 
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FY  2000  PET  Highlights 

Eleanor  Schroeder,  NAVO  MSRC  Program  Environment 
and  Training  Program  (PET)  Government  Lead 


The  Annual  PET  Review  was  held  the 
week  of  April  10th.  As  in  previous 
reviews,  there  was  a  panel  of  external 
reviewers,  consisting  of  people  who 
have  a  broad  background  in  high  per¬ 
formance  computing.  Past  reviewers 
have  included  Kay  Howell  (National 
Coordinating  Office),  Ann  Hayes  (Los 
Alamos  National  Laboratory),  Bob 
Borchers  (National  Science  Founda¬ 
tion),  Valerie  Taylor  (Northwestern  Uni¬ 
versity),  Charles  Bender  (Ohio  Super¬ 
computing  Center),  and  Joan  Novak 
(Environmental  Protection  Agency) . 
Input  from  our  external  reviewers  is 
invaluable  to  our  programmatic  and 
strategic  planning  efforts. 

Our  University  of  Virginia  metacomput¬ 
ing  project.  Legion,  is  progressing  well. 
We  have  been  successfully  interfacing 
with  the  Army  Research  Laboratory 


(ARL)  folks  and  have  set  up  and  run  a 
variety  of  tests.  The  “Legionization”  of 
COAMPS  is  underway  and  on  track. 
Oregon  State  is  progressing  on  its 
Resource  Allocation  Tool.  We  are 
working  through  some  security  issues 
right  now  but  hope  to  have  everything 
resolved  soon.  San  Diego  Supercom¬ 
puting  Center  is  active  on  four  tiger 
team  projects.  For  more  information 
about  these  and  any  of  our  projects, 
visit  our  web  site: 
http  ://www.  navo.  hpc.  mil/pet 

We  hosted  several  workshops  in  the 
first  half  of  this  year.  A  very  successful 
Visualization  Workshop  was  held  in 
January  that  included  participation 
from  all  four  MSRCs.  We  held  a  well- 
attended  seminar  the  day  after  the 
workshop  on  TeraMTA.  We  hosted  the 
third  annual  Signal  Image  Processing 


Workshop,  Unstructured  Grids,  and  a 
Legion  workshop.  More  information 
on  these  workshops  can  be  found  on 
our  web  site  or  by  contacting  Bob 
Melnik  melnik@navo.hpc.mil . 

We  are  currently  soliciting  proposals  for 
our  year  5  efforts.  Please  contact 
Walter  Shackelford 

wshackel@navo.hpc.mil  for  more  infor¬ 
mation.  We  plan  to  involve  our  Com¬ 
putational  Technology  Area  leads  in  the 
review  of  our  proposals  again  this  year. 


Scientific  Visualization  Workshop 

Dr.  Bob  Melnik,  CTA  Coordinator 


NAVO  MSRC  PET  held  its  first  Engineer¬ 
ing  and  Scientific  Data  Visualization  Work¬ 
shop  in  January.  Conducted  in  the  NAVO 
MSRC  PET  classroom  facilities,  the  work¬ 
shop  was  attended  by  26  participants, 
whose  enthusiasm  kept  them  working  well 
into  the  evening  on  their  lab  exercises! 

The  purpose  of  the  three-day  workshop 
was  to  provide  NAVO  MSRC  users  with 
the  information  and  skills  needed  to 
employ  state-of-the-art  visualization  tech¬ 
nology.  Students  were  invited  to  "bring 
your  data"  to  the  workshop,  which  consist¬ 
ed  of  a  mix  of  classroom  lectures  and 
hands-on  laboratory  exercises.  The  Army 
Research  Laboratory  (ARL),  U.S.  Army 
Engineer  Researh  and  Development  Cen¬ 
ter  (ERDC),  and  NAVO  MSRCs  presented 
the  activities  of  their  respective  visualiza¬ 
tion  laboratories  and  participated  in  semi¬ 
nars  on  the  state-of-the-art  in  scientific  and 
engineering  data  visualization  technology. 
The  workshop  concluded  with  a  panel  dis¬ 


cussion  of  emerging  developments  in  visu¬ 
alization  technology. 

For  those  who  could  not  be  present,  work¬ 
shop  materials  can  be  found  at  the  NAVO 
MSRC  web  site: 

http://www.  navo.  hpc.  mil/pe1/viz2000 

Two  visualization  packages,  AVS  5  and 
Data  Explorer,  were  presented  in  tutorials. 
Both  are  full-featured,  state-of-the-art  soft¬ 
ware  systems  that  can  produce  profession¬ 
al  quality  visualizations  of  engineering  and 
scientific  data.  The  tutorials  were  given  at 
an  introductory  level  and  were  directed 
toward  users  with  little  experience  in 
advanced  visualization  software  packages. 

AVS  5  is  a  fully  supported  commercial 
application  for  data  visualization  from 
Advanced  Visual  Systems  of  Waltham, 

MA.  It  is  available  for  both  Unix  and 
Linux  operating  platforms.  Data  Explorer 
is  a  new  open-source  application  that  is 
evolving  from  an  earlier  commercial  appli¬ 
cation  known  as  “IBM  Visualization  Data 


Explorer.”  An  organization  known  as 
OpenDX  has  been  formed  in  cooperation 
with  IBM  to  coordinate  the  development 
of  an  Open  Source  version  of  Data 
Explorer.  Binary  versions  of  Data  Explorer 
for  Windows,  Unix,  and  Linux  platforms 
are  available  for  download,  at  no  charge, 
from  the  OpenDX  organization  web  site. 

Additional  information  on  AVS  5  and 
Data  Explorer  can  be  found  at  their  orga¬ 
nization's  web  sites: 

AVS  5: 

http://www.  avs.com/products/AVS5/ 
avs5.htm 

OpenDX:  http://www.opendx.org 

NAVO  MSRC  would  be  pleased  to  repeat 
the  workshop  in  the  future.  If  you  desire  to 
attend  a  visualization  workshop  in  the 
future,  please  contact  the  NAVO  MSRC 
PET  Training  Coordinator,  Brian  Tabor,  at 
taborb@navo.hpc.mil . 
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PET  Workshops 


2000  Winter  Legion  Workshop 

The  joint  Legion  Group/ Applied 
Metacomputing  Winter  2000  Work¬ 
shop  was  held  in  January  at  the  Uni¬ 
versity  of  Virginia.  Participants 
received  introductory  information, 
started  a  system,  ran  applications,  and 
learned  advanced  system  administra¬ 
tive  tasks  in  hands-on  sessions.  A  sec¬ 
ond  workshop  was  held  in  May.  Both 
workshops  were  well  attended  by  aca¬ 
demic,  government,  and  corporate 
participants  and  included  information 
on  customizing  and  troubleshooting 
systems.  For  more  information  on 
these  workshops,  visit 
http  ://www.  legion .  Virginia .  edu/ 
workshops.html 


Dual-Level  Parallelism  and 
Multi-Block  Grids  and  Coastal 
Ocean  Circulation  Modeling 

This  one  hour  workshop  held  in 
March  introduced  a  multi-block  grid 
generation  technique  and  a  dual-level 
parallel  implementation  into  the 
Princeton  Ocean  Model  (POM).  MPI 
was  used  to  parallelize  the  POM, 
assigning  each  grid  block  a  unique 
processor.  OpenMP  dynamic  thread¬ 
ing  was  used  to  alleviate  workload 
variations  between  MPI  processes.  Dr. 
Phu  Luong,  ERDC  Environmental 
Quality  Modeling  and  Simulation 
(EQM)  On-site  Lead,  demonstrated 
the  efficacy  of  the  MPI-only  and 
MPI/OpenMP  code  versions  of  the 
POM  model  on  both  a  one-block  and 
twenty-block  grids.  For  more  informa¬ 
tion,  visit 

http  ://www.  navo.  hpc .  mil/  cgi-b  in/ pet/ 
Training/training.cgi/ 


Distributed  Computing  and 
Environmental  Modeling 
Workshop 

Mary  Wheeler,  Director  of  The  Center 
for  Subsurface  Modeling  at  University  of 
Texas,  Austin,  and  Phu  Luong,  ERDC 
On-Site  EQM  Lead,  presented  a  com¬ 
puting  portal  for  modeling  multi-physics 
and  multiple  domains  for  environmental 
modeling  at  the  NAVO  MSRC  PET 
training  facility  in  March.  A  computa¬ 
tional  framework  known  as  IPARS  was 
presented  that  permits  rigorous,  physi¬ 
cally  representative  coupling  of  different 
physical  and  numerical  flow  models  in 
different  parts  of  the  domain  on  mas¬ 
sively  parallel  systems  or  clusters  of 
workstations.  For  more  information  on 
this  workshop,  visit 
http  ://www.  navo.  hpc .  mil/cgi-bin/pet/ 
Training/training.cgi/ 


Sun's  High  Performance 
Computing  Programming 

This  four-day  workshop  held  in  March 
presented  programming  techniques  and 
skills  to  optimize  and  parallelize  pro¬ 
grams  for  the  Sun  Ultra  Enterprise  envi¬ 
ronment.  Topics  included  Sun’s  Perfor¬ 
mance  Workshop  Tool  Suite,  Single 
Node/Single  Process  optimization,  SGI 
to  Sun  porting  issues,  automatic  and 
explicit  parallelization,  performance 
analysis  of  parallelized  code,  and  opti¬ 
mizing  and  debugging  MPI  programs. 

In  addition,  an  overview  of  the  Sun 
MPI  and  cluster  runtime  programming 
environment  was  presented.  For  more 
information  on  this  workshop,  visit 
http  ://www.  navo.  hpc .  mil/  cgi-bin/pet/ 
Training/training.cgi/ 
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Tiger  Teams 

Jay  Boisseau,  San  Diego  Supercomputer  Center 


INTRODUCTION 

The  San  Diego  Supercomputer  Cen¬ 
ter  (SDSC)  is  one  of  the  NAVO 
MSRC  PET  academic  affiliates. 

SDSC  has  been  deeply  involved  with 
the  NAVO  MSRC  PET  program  from 
the  very  beginning,  and  the  scope  of 
SDSC's  involvement  has  increased 
each  year  in  the  program.  The  rea¬ 
son  for  SDSC's  involvement  has 
always  been,  and  continues  to  be,  to 
improve  the  computational  infra¬ 
structure  for  scientific  research  at 
both  NAVO  and  SDSC,  thus  making 
users  at  both  sites  more  effective  as 
scientists.  This  article  describes  the 
history  of  this  involvement,  the  rea¬ 
sons  it  has  been  so  successful,  and 
some  of  the  products  of  this  valuable 
partnership.  It  closes  with  some  com¬ 
ments  on  possible  future  interactions 
between  SDSC  and  the  NAVO 
MSRC. 

SIMILAR  MISSIONS  AND 
USERS:  LEVERAGING  AND 
SYNERGY 

Perhaps  the  primary  reason  that  the 
partnership  between  SDSC  and  the 
NAVO  MSRC  has  been  so  effective  is 
that  they  have  similar  missions  and 
goals.  Both  centers  strive  to  increase 
scientific  knowledge  by  making  their 
researchers  more  effective  through 
the  application  of  advanced  comput¬ 
ing  technologies.  Both  have  large 
user  communities  of  world-class  sci¬ 
entists  with  varying  degrees  of  com¬ 
putational  expertise  on  HPC  systems. 
Some  of  the  areas  of  science  even 
overlap:  computational  fluid  dynam¬ 
ics  is  well  represented  in  both  user 
communities,  as  is  research  related  to 
Earth  systems  sciences  (climate/ 
ocean  modeling,  weather  forecasting. 


ground  water  modeling,  etc.).  SDSC 
and  NAVO  MSRC  also  have  similar 
HPC  resources,  and  their  users  face 
similar  hurdles  in  developing  high- 
performance  parallel  applications. 

These  commonalities  have  promoted 
the  development  of  a  unique  rela¬ 
tionship  between  SDSC,  as  an  aca¬ 
demic  affiliate,  and  Logicon  and  the 
NAVO  MSRC.  SDSC,  through  its 
role  as  the  leading-edge  site  in  the 
National  Partnership  for  Advanced 
Computational  Infrastructure 
(NPACI),  and  Logicon,  as  the  inte¬ 
grator  for  the  NAVO  MSRC,  are 
developing  comprehensive  computa¬ 
tional  infrastructures  that  do  more 
than  just  provide  cycles.  Their  goals 
are  to  provide  balanced  archival  stor¬ 
age  hardware  and  networking  capac¬ 
ity,  and  software  technologies.  They 
provide  training  to  master  these  lead¬ 
ing-edge  hardware  and  software 
environments,  which  makes  all  of 
this  easy  to  use  effectively. 


Advanced  Visualization  Systems  (AVS)  User 
Interface 

SDSC  has  conducted  a  number  of 
projects  for  Logicon  and  the  NAVO 
MSRC  (see  next  section)  that  also 
have  value  in  the  NPACI  computa¬ 


tional  environments.  The  NAVO 
MSRC  PET  program  enables  the 
NAVO  MSRC  to  leverage  these 
SDSC/NPACI  technologies,  and  in 
doing  so  accelerates  their  develop¬ 
ment  by  providing  additional/supple¬ 
mentary  funding  to  SDSC  to  devel¬ 
op,  test,  and  implement  these  tech¬ 
nologies.  The  results  of  these  projects 
are  implemented  in  both  environ¬ 
ments  and  therefore  benefit  both  sets 
of  users.  More  importantly,  these 
projects  benefit  more  researchers  on 
a  faster  timescale  than  would  be  pos¬ 
sible  for  either  center  to  accomplish 
on  its  own.  The  resulting  synergy 
enables  both  centers  to  fulfill  their 
missions  more  effectively. 

SDSC  PROJECTS  FOR  NAVO 
PET 

As  stated  above,  SDSC  proposes 
projects  to  Logicon  that  fulfill  the 
mission  of  NAVO  MSRC  PET  and 
also  address  NPACI  strategic  goals. 
Logicon  reviews  these  projects  and 
those  proposed  by  other  academic 
affiliates  for  their  potential  impact  on 
the  capabilities  of  NAVO  MSRC 
users.  Logicon  then  funds  the  proj¬ 
ects  deemed  most  likely  to  contribute 
to  fulfilling  the  NAVO  MSRC  PET 
mission.  In  each  of  the  first  four 
years  of  the  NAVO  MSRC  PET  pro¬ 
gram,  Logicon  has  selected  SDSC  to 
conduct  a  variety  of  projects.  The 
selected  projects  leverage  SDSC's 
expertise  in  a  variety  of  computa¬ 
tional  areas,  and  the  scope  and  fund¬ 
ing  of  these  activities  have  increased 
each  year  due  to  SDSC's  successes. 
Some  of  these  are  noted  in  the  fol¬ 
lowing  paragraphs;  for  more  detailed 
information,  please  refer  to  the 
SDSC  NAVO  MSRC  PET  web  pages: 
http://www.sdsc.edu/PET/NAVO 
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Tiger  Teams 

The  most  prominent  project  initiated 
by  SDSC  and  Logicon  has  been  the 
Tiger  Team  collaborations.  SDSC  and 
Logicon  realized  early  on  that  the  key 
to  helping  scientists  overcome  the 
steep  learning  curves  associated  with 
using  new  systems  and  software  is 
often  intensive,  individual  collabora¬ 
tion  with  computational  experts  who 
have  already  mastered  these  technolo¬ 
gies.  Tiger  Teams  are  collaborations 
between  SDSC  computational  experts 
and  DoD  scientists.  NAVO  MSRC  PET 
Director  Walter  Shackelford,  Compu¬ 
tational  Technology  Area  (CTA)  Liai¬ 
son  Bob  Melnik  of  Logicon,  and 
NAVO  MSRC  PET  Government  Lead 
Eleanor  Schroeder  identify  DoD 
researchers  with  strategic  applications 
for  participation  in  Tiger  Team  collab¬ 
orations,  while  SDSC  NAVO  MSRC 
PET  Lead  Jay  Boisseau  identifies 
SDSC  staff  for  participation.  The 
SDSC  staff  come  from  computational 
science  research  backgrounds,  but 
now  focus  on  the  application  of 
advanced  computing  techniques  to 
solve  problems  on  HPC  systems.  They 
specialize  in  porting  applications  to 
leading-edge  architectures  and  opti¬ 
mizing  the  computation  and  commu¬ 
nications  performance  of  these  appli¬ 
cations.  They  work  with  the  selected 
DoD  scientists,  who  have  already 
made  the  transition  to  using  HPC 
resources  at  the  MSRC  but  who  are 
more  focused  on  the  scientific  tech¬ 
niques  than  on  the  HPC  systems 
themselves.  Thus,  both  parties  are 
familiar  with  the  specialties  of  the 
other,  but  each  brings  expertise  to  the 
table  that  can  be  combined  to  pro¬ 
duce  world-class  scientific  applications. 

The  Tiger  Team  approach  has  already 
achieved  several  notable  successes. 
One  particularly  illustrative  example  of 
the  power  of  this  approach  is  the  Tiger 
Team  project  on  the  CFDSHIP-IOWA 
code.  Bob  Sinkovits  of  SDSC  worked 
with  Eric  Paterson  of  the  University  of 
Iowa  and  his  team  on  optimizing  the 
CFDSHIP-IOWA  application  for 


designing  vessels  on  the  T90  at  NAVO 
MSRC  and  achieved  better  than  a  2 
times  speed-up.  This  would  have  been 
a  tremendous  success  on  its  own 
because  the  code  was  optimized  and 
available  for  scientists  to  conduct  more 
simulations.  Sinkovits  and  Paterson 
carried  the  project  even  further,  how¬ 
ever,  by  teaming  to  develop  a  new 
parallel  version  of  the  code  that 
tremendously  increased  the  capability 
of  future  simulations  in  terms  of  both 
wallclock  performance  and  available 
memory.  The  code  has  since  been 
used  in  areas  from  computational  fluid 
dynamics  to  environmental  modeling 
by  other  researchers  who  had  been 
waiting  for  a  parallel  version  of  this 
code  to  integrate  into  their  research 
projects. 

The  Tiger  Team  approach  has  demon¬ 
strated  many  other  successes  in  the 
past  three  years.  These  successes 
included:  (1)  optimizing  the  numerical 
flow  algorithm  (NFA)  wake  modeling 
code  for  the  NAVO  MSRC  Cray  T3E 
(Stuart  Johnson  of  SDSC  with  Doug 
Dommermuth  of  Science  Applications 
International  Corporation  (SAIC)),  (2) 
optimizing  the  NOGAPS  global  weath¬ 
er  prediction  system  (Giri  Chukkapalli 
of  SDSC  with  Tom  Rosmond  of  NRL- 
Monterey),  (3)  optimizing  the 
dieCAST  regional  ocean  modeling 
code  (Chukkapalli  and  Dongju  Choi 


of  SDSC  with  Steve  Piacsek  of  NRL- 
Stennis),  (4)  developing  a  procedure 
for  optimizing  vector  signal/image  pro¬ 
cessing  library  (VISPL)  routines  for 
IBM  SP  and  Cray  T3E  processors 
(Johnson  with  Rich  Linderman  of 
Wright  Patterson  Air  Force  Base 
(WPAFB)),  and  (5)  providing  external 
reviews  and  guidance  to  the  develop¬ 
ers  of  the  MACH3D  magnetohydrody¬ 
namics  code  (Tim  Kaiser  of  SDSC 
with  Bob  Peterkin  of  Air  Force 
Research  Laboratory  (AFRL))  and  the 
FAST3D  computational  fluid  dynam¬ 
ics  code  (Johnson  with  Jay  Boris  of 
NRL- Washington,  DC).  Current  proj¬ 
ects  include  optimizing  the  CH3D 
ground  water  modeling  code 
(Sinkovits  with  Jeff  Holland  of  ERDC) 
and  the  Coupled  Ocean/Atmosphere 
Mesoscale  Prediction  (CO AMPS) 
regional  weather  prediction  code 
(Chukkapalli  and  Choi  with  Rich 
Hodur  of  NRL-Monterey). 

Thus,  while  SDSC  and  other  NAVO 
MSRC  PET  academic  affiliates  are 
working  in  specific  projects  to  im¬ 
prove  the  ease  of  use  as  well  as  the 
effectiveness  of  the  NAVO  MSRC 
infrastructure  (see  next  section), 

SDSC  and  Logicon  initiated  Tiger 
Teams  with  prominent  DoD  re¬ 
searchers  to  help  them  use  resources 
that  are  currently  still  difficult  to  use 
effectively.  Since  HPC  resources  are 


NAVO  MSRC  Visualization  Lab 

Visualization  and  Data  Analysis  Resource 
Data  Visualization  Builder 


O  Arbitrary  Slice  ^  Volume  Render  ^  Bubble  Vis  ^  Isosurface 


❖  Ortho  Slice  ❖  Threshold  Slice 

VOLUME  RENDER:  The  scalar  value  at  each  grid  location  in  the  3D  dataset  is 
used  as  an  index  into  a  color/opacity  lookup  table.  Each  grid  point  is  then  ren¬ 
dered  in  3D  space  as  a  region  of  the  appropriate  color  and  opacity. 
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always  at  the  cutting  edge,  there  will 
always  be  a  need  for  both  projects 
that  make  the  resources  easier  to  use 
and  for  Tiger  Teams  to  help  re¬ 
searchers  master  the  issues  and  tasks 
that  remain  difficult  to  perform. 

The  Tiger  Team  program  will  be  de¬ 
scribed  in  detail  in  future  issues  of 
Navigator,  including  individual  articles 
on  current  and  future  collaborations. 

Software  Technology  Projects 

Most  SDSC  NAVO  MSRC  PET  proj¬ 
ects  have  been  more  “traditional”  in 
the  sense  that  they  have  funded  the 
development  and  integration  of  soft¬ 
ware  technologies  into  the  NAVO 
MSRC  environment  to  make  it  easier 
for  scientists  to  use  it  more  effectively. 
SDSC,  in  partnership  with  Oregon 
State  University,  provided  the  infra¬ 
structure  to  give  web  access  to  the 
operational  status  and  queues  on  all 
NAVO  MSRC  machines.  This  project, 
led  by  Mary  Thomas  of  SDSC,  Ken 
Ferschweiler  of  Oregon  State,  and 
Andrew  Schatzle  of  Logicon,  enables 
users  to  quickly  determine  which  sys¬ 
tems  are  currently  available  and  which 
queues  are  likely  to  yield  the  best  turn¬ 
around  times  for  jobs.  Thus,  a  simple 
web  page  replaces  several  login  ses¬ 
sions  and  interactive  commands  and 
has  made  using  multiple  NAVO  MSRC 
systems  much  easier.  This  page  is 
available  now  at 
http  ://www.  navo.  hpc.mil/Tools/ 
Queues/. 

Another  web-based  project  designed 
to  make  MSRC  facilities  easier  to  use 
is  PCOMP  the  Parallel  Computing  Por¬ 
tal.  PCOMP  led  by  Laura  Nett  of 
SDSC,  is  designed  to  help  parallel 
application  developers  quickly  find  any 
source  of  general  information  on  par¬ 
allel  computing,  from  web  pages  on 
programming  tools  (message  passing 
interface  (MPI),  High  Performance  For¬ 
tran  (HPF),  etc.)  to  vendor  documen¬ 
tation  (Cray  T3E,  Sun  HPCIOOOO, 

IBM  SP  etc.),  to  benchmarks,  to  HPC 
journals  and  newsletters.  PCOMP 
complements  the  site-specific  docu¬ 


mentation  provided  by  NAVO  MSRC 
PET  to  assist  the  users  with  specific 
MSRC  systems. 

Scientific  visualization  is  an  important 
part  of  computational  science,  and 
SDSC  has  developed  two  projects 
designed  to  make  MSRC  visualization 
facilities  easier  to  use.  One  is  the  Visu¬ 
alization  and  Data  Analysis  Resource, 
VADAR,  developed  by  Greg  Johnson 
of  SDSC.  It  helps  researchers  learn  to 
use  the  MSRC  Visualization  Center 
facilities  to  visualize  their  data  in  order 
to  gain  insight  into  the  results  of  their 
simulations.  It  introduces  them  to  the 
hardware  and  software  available  in  the 
Vislab  and  guides  them  to  the  correct 
choices  for  the  visualization  needs.  It 
then  steps  them  through  the  process  of 
visualizing  their  data,  based  on  a  series 
of  simple  questions.  VADAR  is  avail¬ 
able  to  NAVO  MSRC  users  in  the 
MSRC  Visualization  Center.  The  other 
web-based  project  designed  to  make 
the  MSRC  Visualization  Center 
resources  easier  to  use  is  the  collabora¬ 
tive  extension  to  the  AVS  available  at 
http  ://www.  navo.  hpc .  mil/cgi-bin/ 
pei/Training/Resources/ 
resources2 . cgi?resource = avs,  also 
developed  by  Greg  Johnson.  This  tool 
makes  it  possible  for  researchers  to 
explore  their  data  visually  over  the 
Internet,  sharing  views  of  the  visualiza¬ 
tions  that  each  party  has  the  capability 
to  alter.  This  can  increase  the  tremen¬ 
dous  productivity  of  colleagues  at  dif¬ 
ferent  locations  working  on  the  same 
project.  Together,  these  tools  enhance 
the  overall  productivity  of  MSRC  users 
by  enabling  them  to  gain  insight  into 
their  simulations  more  easily  and  rap¬ 
idly. 

An  area  of  especially  active  collabora¬ 
tion  between  SDSC  and  NAVO  MSRC 
has  been  security  infrastructure.  SDSC 
has  long  been  recognized  as  a  center 
of  expertise  in  security  technologies. 
The  DoD  MSRCs  implement  security 
measures  to  protect  the  integrity  of  the 
systems  and  of  users'  codes  and  data. 
Thus,  this  was  a  natural  area  of  part¬ 
nership  and  has  been  funded  every 


year  by  Logicon.  Tom  Perrine  of 
SDSC,  a  nationally  recognized  leader 
in  computer  security  technologies,  has 
worked  on  numerous  security  issues 
and  projects,  delivering  expertise, 
white  papers,  and  tools  to  the  staff  at 
the  NAVO  MSRC  to  provide  a  secure 
computing  environment. 

Finally,  SDSC  has  contributed  to  a 
number  of  system-related  projects  that 
increase  the  functionality  and  robust¬ 
ness  of  NAVO  MSRC  systems.  At  the 
request  of  Northrop-Grumman,  SDSC 
led  a  workshop  for  sites  with  large 
Cray  T3Es  to  discuss  a  variety  of  sys¬ 
tem  performance,  security,  and  reliabil¬ 
ity  issues.  SDSC  has  responded  to  a 
number  of  other  requests  for  advice  on 
systems  issues  from  NAVO  MSRC 
staff.  SDSC  also  participated  in  the 
porting  and  optimization  of  Legion  for 
HPC  systems,  which  has  now  been 
transferred  to  the  MSRC  staff. 

Training 

SDSC  has  participated  in  the  NAVO 
MSRC  PET  Training  Program  as  well 
as  developing  software  technologies. 
An  extremely  comprehensive  training 
program,  led  by  Brian  Tabor  and  Bob 
Melnik  of  Logicon,  teaches  users  how 
to  use  MSRC  facilities  more  effectively. 
SDSC  has  been  asked  to  teach  courses 
in  parallel  computing  and  in  scientific 
visualization  as  part  of  this  training 
program.  Some  of  these  classes  are 
available  for  viewing  on  the  Internet 
from  the  NAVO  MSRC  PET  Training 
web  pages  http://www.navo.hpc.mil/ 
cgi-bin/pei/Training/training.cgi  .  SDSC 
classes  have  been  very  well  received, 
and  the  visualization  classes  in  particu¬ 
lar,  taught  by  SDSC's  Greg  Johnson, 
are  in  high  demand. 
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Supporting  the  Supercomputers 

Part  1  of  2 

By  David  Magee,  Senior  Systems  Manager 


In  the  supercomputing  arena,  the  sta¬ 
tistic  that  generally  earns  the  largest 
headline  is  the  peak  FLOPS  rating  for 
a  given  system  or  center.  However, 
successful  implementation  of  multi- 
teraflop  HPC  systems  and 
centers  requires  a  clear  bal¬ 
ance  of  computational, 
mass  storage,  and  net¬ 
working  capabilities — com¬ 
puting  power  has  little 
value  if  commensurately 
robust  storage  and  net¬ 
works  are  not  available. 

By  the  end  of  calendar 
year  1999,  the  Cray  J916- 
based  mass  storage 
archival  server  at  the 
NAVO  MSRC  was  manag¬ 
ing  in  excess  of  160  ter¬ 
abytes  of  on-line  storage. 

Current  projections  show  a 
rapid  expansion  during 
calendar  year  2000, 
approaching  400  terabytes  by  year's 
end!  The  planned  upgrade  of  the 
Cray  T3E  to  1  teraflop  and  the  addi¬ 
tion  of  a  2-teraflop  IBM  SP  require 
that  we  enhance  our  mass  storage 
and  network  infrastructure  to  permit 
the  most  effective  use  of  these  sys¬ 
tems,  primarily  by  DoD  Challenge 
projects  and  users. 

In  mid-1999,  the  NAVO  MSRC 
began  implementing  significant 
improvements  in  its  infrastructure  in 
two  concurrent  projects.  The  first 
project  was  to  design  a  resilient  mass 
storage  server  capable  of  meeting 
current  demands  with  the  ability  to 
scale  and  accommodate  projected 
growth  rates  which  will  exceed  1  ter¬ 
abyte  of  new  data  per  day  with  total 
archive  capacities  measured  in 
petabytes.  The  second  project  was  to 
reengineer  the  MSRC  internal  net¬ 


work  to  incorporate  state-of-the-art 
networking  technologies  that  provide 
unprecedented  performance  and 
resiliency. 


will  include  the  new  STK  9840  family 
of  high-density,  high-performance 
tape  drives.  In  the  event  of  a  compo¬ 
nent  (system,  network,  disk,  tape 
drive,  etc.)  failure,  Veritas  Cluster 

software  will  facilitate  the 
automated  failover  from 
one  ElOK  system,  compo¬ 
nent,  or  service  to  another, 
ensuring  high  availability 
and  minimizing  service 
interruptions  to  users.  We 
believe  this  RMSS  configu¬ 
ration  will  provide  NAVO 
MSRC  users  with  a  scala¬ 
ble,  robust,  resilient  mass 
storage  server,  using  lead¬ 
ing-edge  technology  to 
keep  pace  with  the  grow¬ 
ing  demand  for  data  stor¬ 
age  and  retrieval.  This 


Two  Sun  Microsystems, 

ElOOOOs  (ElOKs),  were 
selected  as  the  core  hard¬ 
ware  platform  for  the 
resilient  mass  storage 
server  (RMSS).  Storage 
and  Archive  Manager/ 

Quick  File  System  (SAM- 
QFS)  from  LSC,  Inc.,  was 
selected  to  provide  a 
high-performance  file  sys¬ 
tem  with  robust  and  full- 
featured  hierarchical  storage  manage¬ 
ment  capabilities.  The  ElOKs  share  a 
large  multi-terabyte  array  of  fibre 
channel  Redundant  Array  of  Inex¬ 
pensive  Disks  (RAIDs)  capable  of 
transfer  speeds  up  to  100  megabytes 
per  second  per  individual  array.  Mul¬ 
tiple  StorageTek  (STK)  robotic  tape 
libraries  are  used  to  house  the 
archive  tape  media  and  drives,  which 


same  configuration  is  being  imple¬ 
mented  at  other  DoD  MSRCs  and  is 
expected  to  form  the  basis  for  a 
robust,  common,  wide-area  RMSS 
capability  across  the  DoD  Shared 
Resource  Centers. 

In  the  next  issue.  Part  2  of  this  article 
will  describe  the  ongoing  NAVO 
MSRC  network  upgrades. 
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T3E  Scalability 

Dave  Cole,  NAVO  MSRC  Computer  Systems  and  Support 


DoD-wide  demand  for  the  NAVO  MSRC  Cray  T3E  system  has  been  and  remains  extremely  high,  making  it 
the  most  requested  and  successful  scaleable  parallel  HPC  system  in  the  DoD  HPC  Modernization  Program. 
In  order  to  evaluate  the  T3E's  ability  to  process  even  larger  individual  applications  and  aggregate  workloads, 
it  was  temporarily  upgraded  in  late  November  1999  from  816  to  1,088  processors  with  an  aggregate  mem¬ 
ory  of  approximately  400  gigabytes.  This  configuration  positioned  NAVO’s  T3E  as  the  largest  unclassified 
T3E  in  the  world  and  number  8  in  the  Top  500  ranking. 


The  evaluation  of  the  expanded  T3E 
configuration  ended  in  early  February 
2000  and  was  conducted  in  two  phas¬ 
es.  Phase  1  utilized  existing  challenge 
and  non-challenge  work  to  test  aggre¬ 
gate  workload  scheduling  with  the  pri¬ 
mary  goal  of  demonstrating  a  substan¬ 
tive  improvement  in  system/job 
throughput  as  evidenced  by  a  reduction 
in  the  general  "expansion  factors"  for 
batch  queues  and  queue  complexes. 

No  user  action  or  intervention  was 
required  during  this  period  of  time. 

Phase  2  consisted  of  scalability  tests  for 
selected  applications  that  could  effectively  utilize  much 
larger  shares  of  T3E  processor  and  memory  resources. 
Phase  2  invitations  were  extended  to  all  Computational 
Technical  Area  (CTA)  leaders  and  to  all  DoD  Challenge 
Project  Principal  Investigators  (Pis)  who  were  granted  an 
FY2000  allocation  on  any  DoD  HPC  system. 

Four  challenge  projects  representing  three  CTAs  and  two 
Services  participated  in  the  Phase  2  scalability  tests  and 
are  listed  in  the  table  below.  Members  of  the  first  two 
projects  were  unable  to  scale  their  code  during  the  test 
period.  The  results  from  the  two  projects  which  success¬ 
fully  utilized  the  expanded  T3E  configuration,  with  com¬ 
ments  by  the  respective  Pis  and/or  authors,  are  available 
in  the  NAVO  MSRC  T3E  Scalability  Report. 

Our  analysis  of  Phase  1  utilization  data  for  the  months  of 
October  (before)  and  December  (after)  documents  an 
increase  in  the  number  of  jobs  run,  the  capability  to  run 


large  jobs,  and  better  throughput  as 
indicated  by  a  decrease  in  the  Normal¬ 
ized  Expansion  Factor.  Phase  2  scala¬ 
bility  test  applications  demonstrated  the 
capability  of  the  enhanced  T3E  to  sup¬ 
port  world-class  high  performance  com¬ 
puting.  A  1/32-degree  near-global  simu¬ 
lation,  which  typically  uses  474  proces¬ 
sors,  was  run  on  948  processors.  The 
decrease  in  run  time  from  14.35  hours 
to  8.35  hours  achieved  a  near  linear 
speed  up  of  the  finest  resolution  global 
ocean  model. 

Benchmark  tests  using  1,024  processors 
included  1.02-biIIion-atom  Molecular  Dynamics  simula¬ 
tion  and  22,500-atom  Quantum  Molecular  Dynamics 
simulations  within  the  Density  Functional  Theory — ^the 
largest  classical  and  quantum  molecular-dynamics  sys¬ 
tems  simulation  in  history.  Additionally,  aggregate  work¬ 
load  handling  was  vastly  improved,  thereby  demonstrat¬ 
ing  a  capacity  solution  in  addition  to  an  unparalleled 
DoD  HPC  single  system  capability.  T3E  usage  during  the 
test  period  is  a  clear  indicator  of  the  need  for  scaleable 
parallel-distributed  memory  HPC  systems  of  this  size. 

The  NAVO  MSRC  gratefully  acknowledges  the  hardware, 
software,  and  support  provided  by  SGI  and  Logicon 
Inc.,  which  made  the  T3E  test  and  evaluation  possible. 
We  would  also  like  to  thank  all  Challenge  team  members 
who  participated  in  the  Phase  2  scalability  tests  for  their 
time  and  efforts. 


CTA 

Service 

CFD 

Air  Force 

CCM 

Air  Force 

CCM 

Air  Force 

CWO 

Navy 

Project 

Unsteady  Aerodynamics  of  Aircraft  Maneuvering  at  High  Angles  of  Attack 
Design  of  New  Materials  Using  Computational  Chemistry 
Computational  Assisted  Development  of  High  Temperature  Structural  Materials 
Global  and  Basin-Scale  Ocean  Modeling  and  Prediction 
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Look  Inside  NAVO  MSRC 


We  welcome  our  visitors... 


Dr.  BuzzAldrin,  Gemini  and 
Apollo  Astronaut,  receives  a 
tour  from  Terry  Blanchard, 
NAVO  MSRC  Deoutv  Director 


George  Walker,  Chairman  of  the  State  Board  for  Junior  and  Community  Col¬ 
leges;  Lois  McMurchy,  Director  of  the  Skill  Technology  Center,  Coahoma  Com¬ 
munity  College;  and  RADM  Paul  G.  Gaffney,  Chief  of  Naval  Research,  visit  the 
NAVO  MSRC  booth  at  the  JASON  XII  Exhibit  at  Coahoma  Community  College, 
Clarksdale,  Mississippi 


Challenge  Researchers,  Eric 
Chassignet  and  Remy  Baraille, 
receive  a  tour  of  operations  from 
Scientific  Visualization  Staff,  Pete 
Gruzinskas  and  Andy  Haas 


Left  to  right:  Ms.  Susan  Butler,  Legislative  Staff  Member;  CAPT  Larry 
Icommanding  Officer,  Naval  Oceanographic  Office;  Dr.  Vance  Coffman,  Chief  Executive 
Officer,  Lockheed  Martin;  Mr.  Eric  Womble,  Legislative  Staff  Member;  Congressman 
Chip  Pickering,  3rd  District,  State  of  Mississippi;  Mr.  Jim  Sartucci,  Legislative  Director 
for  Senator  Trent  Lott;  Sonny  Montgomery,  Former  Congressman,  State  of  Mississippi; 
RADM  Kenneth  Barbor,  Commander,  Naval  Meteorology  and  Oceanography  Command 


Steve  Adamec,  NAVO  MSRC  Center 
Director,  and  Congressman  Chip 
Pickering,  3rd  District,  State  of  Mississippi  ^ 
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Navigator  Tools  and  Tips 


Quotas 

Sheila  Carbonette,  User  Services 

Users  are  allocated  a  home  directory  on  each  NAVO  MSRC  system  with  a  certain  amount  of  permanent  non-migrat- 
ed  storage.  Disk  quotas  on  these  home  directories  are  enforced  to  prevent  the  system’s  disks  from  becoming  full.  The 
current  home  directory  quotas  are  listed  below: 

Maximum  Size  of  $HOME 

System  (in  megabytes) 

T3E  (seymour)  250 

T90  (neptune)  250 

SV-1  (poseidon)  250 

02K  (odyssey)  500 


The  “quota  -v”  command  can  be  used  to  display  users’  disk  usage  and  limits  even  if  they  have  not  been  exceeded.  The  out¬ 
put  from  this  command  ran  on  the  T3E  (seymour)  and  Origin  2000  (odyssey)  will  look  similar  to  the  following  example  for 
a  UNICOS  system: 


seymour%  quota  -v 
File  system:  /u/home 
User:  shecar,  Id:  214 

File  blocks  (512  bytes)  Inodes 


User  Quota:  512000*  ( 69.2%)  Unlimited* 

Warning:  460800*  (  76.8%)  None 

Usage:  354048  928 

The  “File  blocks”  column  shows  the  quota  block  counts  and  block  usage  for  online  quotas.  All  values  are  measured  in  512- 
byte  blocks.  The  “Inodes”  column  relates  to  inode  quotas  and  these  quotas  are  not  implemented  on  the  UNICOS  systems. 
The  “User  Quota”  row  shows  a  user’s  current  disk  quota.  The  “Usage”  row  shows  how  much  disk  space  the  user’s  home 
directory  is  currently  using.  The  following  is  an  example  for  an  IRIX  system: 


odyssey%  quota  -v 

Disk  quotas  for  shecar  (uid  214): 


Filesystem 

usage 

quota 

limit 

timeleft 

files 

quota 

limit 

timeleft 

/tmp 

36 

0 

0 

11 

0 

0 

0 

0 

/scr 

8 

0 

0 

18 

0 

0 

0 

0 

/u/home 

511880 

512000 

512000 

1699 

0 

0 

0 

0 

The  “usage”  column  shows  how  much  disk  space  the  user’s  home  directory  is  currently  using.  The  “quota”  column  shows  a 
user’s  current  disk  quota.  The  “limit”  column  shows  the  maximum  amount  of  disk  space  a  user’s  account  can  use.  The  remain¬ 
ing  columns  relate  to  inode  quotas,  and  these  quotas  are  not  implemented  on  the  IRIX  systems.  All  values  are  measured  in 
kilobytes.  The  /tmp  and  /scr  filesystems  are  for  information  purposes  only.  Quotas  are  not  implemented  on  these  systems. 
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Upcoming  Events 


June  4-7,  2000 

User’s  Group  Conference 
Albuquerque,  NM 

June  19-22,  2000 

Government  Technet  2000  Conference 
DC  Convention  Center 
Washington,  DC 

July  23-28,  2000 
SIGGRAPH  2000 
New  Orleans,  LA 
WWW.  siggraph .  org/ 


November  4-10,  2000 
Supercomputing  ‘00 
Dallas,  TX 
www.sc00.org/ 

November  28  -  December  2,  2000 

IEEE  International  Conference  on  Cluster 
Computing  CLUSTER  2000  Germany 
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